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The direction of university courses is often guided by the results of traditional Likert scale student 
evaluations. Most of these focus on instructors’ characteristics and frequently do not provide useful 
insights into students’ learning preferences or feedback regarding specific activities and projects in 
the courses. This study, carried out in a Midwestern U.S. university, reports the use of Q 
methodology to capture students’ views of 35 activities in a graduate TESOL (Teaching English to 
Speakers of Other Languages) Methods course as to which were most helpful and least helpful to 
learning course content. During the last class of the semester 19 students carried out Q sorts about 
different aspects of the methods course. Factor analysis showed how participants grouped onto 3 
factors, expressing 3 unique views on how helpful the 35 different course activities were to their 
learning. The majority of the students were “group-centered learners” who learned best through 
various face-to-face interactions with classmates. Two were “self-centered learners” who learned 
best by working independently, then receiving feedback. One learned best through the course’s 
online activities. Analysis of students’ different views helped researchers determine whether to 
redesign various aspects of the course to meet different learning preferences. 


Universities face many challenges in their desire to 
offer high quality courses that meet the needs of a wide 
array of learners. In an effort to collect student 
feedback, summative course evaluations have been 
essential elements of most college courses for many 
years. At some point in a university course, students 
typically complete evaluations which provide data 
regarding various features such as course design, 
quality of teaching, use of resources, and overall 
effectiveness (Jurczyk & Ramlo, 2004). According to 
Frick, Chadha, Watson, and Zlatkovska, (2010), 
“Course evaluations traditionally used in higher 
education have few items that are empirically related to 
student learning” (p. 116). These items yield few 
insights into student learning preferences or individual 
paths to achievement. An issue in the use of student 
rating for instructional improvement is that the 
feedback is often based on general items or is focused 
on the instructor, making it difficult to use data to 
improve the course (Pohlman, 1975). Indeed, higher 
education institutions have placed heightened attention 
on the need for alternate forms of student course 
evaluation (Amin, 1993). 

Using Q methodology as part of course feedback 
can provide a vehicle for gathering students’ 
opinions and uncovering insights into their views of 
the course and how it supports their learning, aspects 
often missed in traditional Likert scale course 
evaluations (Jurczyk & Ramlo, 2004). As noted by 
Brown (1986), the instrumental basis of Q 
methodology is the Q sort technique, which 
conventionally involves the rank ordering of a set of 
statements from most unlike my view to most like my 
view. Therefore, this methodology provides the way 
for uncovering and identifying the range of 
participant opinions regarding a specific course. 


This mixed method study investigated the views of 
students enrolled in a TESOL (Teaching English to 
Speakers of Other Languages) methods and materials 
course regarding which aspects and activities of the 
course best helped them learn course content. The study 
utilized Q methodology, commonly referred to as Q, 
which provided the means for uncovering and 
identifying the range of participant opinions regarding a 
specific topic of investigation (Stephenson, 1953). 

Literature Review 

The importance of preparing teachers to address 
the needs of English language learners (ELLs) has been 
well documented nationwide but became acutely aware 
to the instructors of the course under study as a result of 
a survey conducted within the Midwestern state where 
this study was carried out. Responses to a teacher 
survey related to professional development research in 
the state where this study took place showed that 96% 
of teachers responding had ELLs in their classrooms 
(Newman, Samimy & Romstedt, 2010). This 
underscored to the researchers in this study the urgent 
need for high quality offerings for teachers in the state 
to learn about identifying and meeting the needs of 
English language learners. For this reason, researchers 
determined that finding out via a learner centered 
course evaluation which learning aspects and which 
activities were most helpful to these graduate students’ 
learning would be the best way to find out in what ways 
the course matched their learning preferences. 

Different approaches to evaluating courses have 
been suggested over the years with the argument that 
they provide clearer insights into identifying how 
students evaluate their courses and instructors. For 
example, Patrick (2011) has written that the five basic 
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instructor personality traits students rate strongly in 
their evaluations are neuroticism (emotional stability), 
extraversion, openness, agreeableness, and 
conscientiousness. Instructor openness and 
conscientiousness added significantly to students’ 
ratings of their courses and professors’ effectiveness. 

Many universities, including the one where this 
study was conducted, use Likert scale course 
evaluations to gather students’ overall impressions, but 
those generally provide few insights into aspects of the 
courses that support learning or specific dimensions 
that students find useful. Other sources of student 
feedback on courses may also include checklists. While 
instructors design courses based upon their ideas about 
how to make them effective, overall, little is known 
about the effects on learning that arise due to 
differences that exist between perceptions of learning 
held by university instructors and students (Lecouteur 
& Delfabbro, 2001), and researchers were looking for 
more information they could use to guide future 
presentations of the course under study. Recently, a 
novel approach for investigating students’ opinions of 
different aspects of a given course has been used by 
several researchers (Jurczyk & Ramlo, 2004; Ramlo, 
2008; Wheeler & Montgomery, 2009). This approach 
uses Q methodology which is considered “an 
appropriate choice whenever a researcher wishes to 
determine the various perspectives and consensus 
within a group regarding any topic” (Ramlo, 2008, p. 
77). In a Q study, participants are presented with a 
number of statements about some topic (in this case, the 
helpfulness of different aspects and activities of a given 
course) and are asked to rank order them according to 
their personal opinion or feelings about these 
statements. The individual rankings of the different 
people’s viewpoints are then analyzed using factor 
analysis (For explicit details of Q Methodology please 
see the “Description of Q Methodology” section). Q 
Methodology (often referred to as Q) was determined to 
be the best choice for this study in uncovering different 
points of view regarding the helpfulness of different 
course aspects and activities to the students. Q is an 
established research methodology used to study 
participants’ subjectivity, or viewpoints, in a systematic 
way (Brown, 1991; McKeown & Thomas, 1988). 
Developed by Stephenson in 1935, Q has served as the 
foundation for the scientific study of subjectivity and 
showed that people’s thoughts can be studied. Q is an 
appropriate choice when studying the perspectives of a 
small number of participants as they rank order a 
number of statements that pertain to the topic under 
study (Brown, 1986, 1991; Stephenson, 1953). 

Jurczyk and Ramlo (2004) noted that using Q as a 
course evaluation leads to “a more complete 
understanding of students as individuals compared to 
the traditional Likert scale course evaluations” (p. 14). 


In a subsequent study, Ramlo (2008) used Q to uncover 
students’ views of their learning in a college physics 
course relative to the instructor’s perceptions of 
students’ typical and ideal views. While the majority of 
the student viewpoints aligned with those of the 
instructor, another perspective coming from Ramlo’s 
study emphasized the need to understand how to 
improve his learning while a third perspective indicated 
that learning the content would either be immediate or 
unlikely. Overall, students in that study indicated that 
they did not utilize the professor’s chosen textbook 
much, if at all. 

Lecouteur and Delfabbro (2001) used Q 
methodology to compare instructors’ and students’ 
views of learning and discovered that “there was not a 
great deal of similarity in the accounts of teaching and 
learning produced by. . .teachers and students in this 
study” (p. 226). Those authors recommended further 
study of available repertoires of teaching and learning 
and how they apply to local contexts. 

When community college students used Q to 
describe their views on learning math, they expressed 
their distinct perspectives through the relative 
placements of Q sort items and open-ended follow-up 
questions based upon their sorts more consistently than 
was possible using available Likert scales (Wheeler & 
Montgomery, 2009). Nevertheless, despite their three 
distinct views, students concurred that the teacher was 
the most important factor in learning math, a conclusion 
subsequently supported by further research. In a study 
involving teachers in a professional development 
workshop, Ramlo (2012a) determined that “Q 
methodology can provide a means of determining 
holistic views about learning at any point during the 
workshop” (p. 7). These workshop participants were all 
in-service teachers, and the authors in the current study 
generalized this observation to determine that Q would 
be an appropriate venue to explore perspectives of adult 
learners (mostly practicing teachers) enrolled in the 
graduate-level course under study. 

With the intent of identifying the versatility of Q in 
higher education, Ramlo (2012b) wrote about three 
studies which underscore the usefulness of Q 
methodology. One study grouped students according to 
their views as they evaluated a newly developed course. 
Another study investigated students’ views of their 
learning in a physics course, a study that replicated 
previous investigations. Results showed similar results 
in the studies, which used the same Q sort items each 
time. The third study investigated faculty perspectives 
as they carried out professional readings and 
professional development in the formation of a new 
school of technology at the university. Consensus and 
distinctions among faculty perspectives were 
highlighted in an inclusive setting. These three studies 
underscored the advantages of Q over traditional 
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surveys and Likert scales in collecting feedback and 
ascertaining individual participants’ views and 
improving overall course quality. 

Description of Q 

A Q study begins by gathering a concourse, which 
is a collection of statements related to the topic under 
study. Imagine a professor interested in students’ 
subjective opinion of her class. She wants to know 
which aspects of the class and which activities that 
students were engaged in during the semester were 
most helpful to them. The professor writes a list of all 
activities and adds statements related to different 
aspects of the class. For example: methods quiz, 
feedback from the professor, student-student 
interaction, or my desire for a good grade. From this 
concourse, the researcher selects a Q sort which is 
placed on a set of numbered cards, each card showing 
an individual statement selected from the concourse. 
Participants force rank these individual items, 
frequently from (-5) most unlike my view to (5) most 
like my view. Ranking decisions are recorded on a Q 
grid data sheet, similar to Figure 1. 

Q methodology is unique, and it differs from 
survey research even though both methods uncover 
participants’ perceptions or viewpoints. Unlike surveys 
and Likert scales, Q sorting ensures that participants 
make explicit choices by ranking of each of the sort 
items relative to the other items while discriminating 
among them in a way they would not do otherwise 
(Corr, 2001; McKeown & Thomas, 1988; Ramlo, 
2008). Such distinctions occur since using a forced 
distribution in the sorting process limits the number of 
items that participants place at each ranking level. 
Another unique feature of Q is that it can be 
accomplished and be effective with small numbers of 
people (Brown, 1986, 1991; Stephenson, 1953). The 
intense nature of Q methodology calls for small 
numbers of participants, or even single case studies, in 
order to explore the existing viewpoints and make them 
open to study (McKeown & Thomas, 2013). 

An appropriate and adequate number of 
participants in a Q study could be small but must 
include enough participants to establish that a varied 
number of points of view exist regarding the topic 
under study (Watts & Stenner, 2012). Participants 
should be selected thoughtfully to be sure to incorporate 
individuals who have specific and relevant opinions on 
the topic (McKeown & Thomas, 2013). 

Through Q methodology, operant categories are 
identified that represent functional, not just logical 
distinctions (Brown, 1991) among participants’ 
perspectives. Capturing different perspectives allows a 
researcher to “understand a human experience rather 
than identify cause-and-effect relationships” (Broady- 


Ortman, 2002, p. 110), while finding out different 
opinions of group members and how many people in 
the group share specific opinions (McKeown & 
Thomas, 2013; Ramlo, 2008; Stephenson, 1953). 

Q methodology follows subjective perspectives 
patterns across participants rather than patterns across 
variables. Thus, Q reveals correlations and factors 
among persons and their views while survey research 
reveals correlations and factors among traits. In Q, the 
correlations are based on the assumption that “persons 
significantly associated with a given factor ... share a 
common perspective” (McKeown & Thomas, 1988, p. 
17). 

Method 

Setting 

The setting for this study was fall semester in a 
graduate level course in TESOL methods at a large 
Midwestern urban university. Since it was the 
beginning of the school year, researchers recognized 
that course content would be available for teachers to 
apply to instruction in the current academic year, 
enabling study participants to apply and implement 
course content immediately and judge which course 
activities were most helpful. 

This study was carried out by two researchers. One 
was the course developer/instructor, and the other was 
the instructor who taught the section of the course in 
this study. Their intent was to obtain the students’ 
views of different aspects and activities in a graduate 
TESOL course to find out which were the most helpful 
and least helpful ones to learning the course content. 
Additionally, this was the first time this course was 
offered as a blended course (with some online content 
and discussion), and the researchers wanted to find out 
whether the students found the online components to be 
helpful as well. Currently, this is the only course in this 
university’s TESOL endorsement program that does not 
have an online version. This is considered the 
cornerstone course in the program, so, based upon 
students’ feedback, researchers intended to use the 
results of this study to aid in a possible redesign of the 
course to make it more meaningful for the students. The 
research question for this study was: What are students’ 
views of the different aspects and activities in the 
TESOL methods course in terms of their helpfulness to 
their learning? 

Description of the TESOL Methods and Materials 
Course 

This course was designed to offer students 
opportunities to develop both content and experiential 
knowledge in teaching English language learners 
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Least helpful to my learning 


Figure 1 

Sort Distribution Grid 


Most helpful to my learning 



(ELLs). Through lectures, readings, class discussions 
and class activities, the course provides a critical 
exploration and analysis of current approaches to 
language teaching with an emphasis on the 
development of communicative competence. During the 
first half of the course students prepared group 
presentations on one of eight methods for teaching 
ESL, and they took a quiz on the different methods 
studied. 

The second purpose of the course was to offer 
students opportunities to gain experiential knowledge 
through teaching mini-lessons. Each student chose one 
of six skills (reading, writing, speaking, listening, 
grammar, or vocabulary) as the focus of a mini-lesson. 
The mini-lessons were taught to the instructor and other 
students who acted in the role of ESL students. All 
observers provided feedback on each mini-lesson. 
Students also wrote personal reflections on their own 
lesson presentations. The final project for this class was 
to write a personal teaching philosophy paper regarding 
teaching ELLs. 

Participants 

Nineteen students (14 females and 5 males) who 
were enrolled in the TESOL methods course 
participated in this study and included one 
undergraduate and 18 graduate students. Eleven of 
those in the class had or were seeking licensure for 
elementary grades (up to grade eight) and seven for 
secondary or K-12, while one participant was preparing 
to work with adults. Fifteen students cited this course as 
a requirement, and four did not. While all were 
preparing to work with English language learners, their 
experience working with ELLs over the past two years 


varied. Seven had worked with nine ELLs or fewer, one 
had worked with 10-19, four had worked with 21-29 
ELLs, two worked with 30-49 ELLs, and five had 
worked with more than 50 ELLs over the past 2 years. 
The amount of previous preparation for working with 
ELLs varied as well. Seven participants had had no 
prior course work in the area of teaching English to 
speakers of other languages. Four had completed one or 
two courses prior to this one, and six participants had 
completed three to five courses. Two participants had 
completed six or more courses and were nearly at the 
end of their TESOL endorsement program. 

Instrument 

An essential aspect of any Q study is the selection of 
the Q sample, or Q sort, which refers to the items selected 
by the researcher to be ranked by participants during the 
sorting process. In this study, the concourse consisted of all 
course activities as well as statements reflecting the 
students’ personal perspectives like student motivation 
regarding the course. The Q sort was generated by the 
researchers from the course syllabus and literature reporting 
on other studies which investigated college courses using Q. 
Researchers started with 42 course related activities and 
aspects and after reviewing them selected 35 sort items for 
the final Q sort, listing the statements on separate cards. The 
sort included all activities and tasks completed throughout 
the semester (see Appendix) and some personal 
perspectives of students. 

Data Collection 

Data was collected through Q sorts (Figure 1) 
which were completed by each of the 19 participants 
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during the last class of the fall semester. During the 
sorting activity, students force ranked 35 statements 
into a quasinormal, symmetrical distribution to 
prioritize among the sort items for the research question 
(Figure 1). Participants sorted the numbered cards from 
least helpful to my learning to most helpful to my 
learning and then copied the numbers from their 
completed sort to a grid sheet. Then they answered 
follow-up questions explaining their sorting decisions 
for the highest (5, 4) and lowest (-5, -4) ranked 
statements along with demographic questions. 

After ranking the statements, the participants were 
asked for follow up explanations of specific rankings 
which were used as qualitative data that provided 
insights into the various ways that different viewpoints 
were represented among participants (Corr, 2001). One 
researcher (also the course instructor) conducted the Q 
sort activity, facilitating the sorting process by 
explaining the data collection process to the whole 
class. Participation was voluntary, as details of the data 
collection process were shared with students a week 
before the sorting activity was scheduled to take place, 
giving students the option of not participating during 
the upcoming class meeting if they did not want to do 
so. All participants freely signed letters of consent, 
indicating that they were participating voluntarily. On 
the designated day, students sorted at their desks 
independently and submitted their grid sheets with no 
names into an envelope. To eliminate bias, students 
were assured that no data would be reviewed or 
analyzed until after final grades had been submitted. 
Responses were kept confidential as results were 
analyzed and reported. 

Students sorted the cards first in three piles: (a) the 
aspects and activities they thought most helped them 
learn course content, (b) those they thought least helped 
them learn, and (c) the ones about which they felt 
neutral. Then they distributed the statements according 
to the individual cells of the sorting grid (Figure 1) and 
answered demographic questions as well as questions 
about their sorting decisions by writing brief 
explanations for the items they ranked as -5, -4, 5, and 
4. They also had the opportunity to provide comments 
about the overall sorting process and the Q sample. 

Data Analysis 

The 19 Q sorts generated in this study were 
analyzed using the PQMethod 2.11 software program 
(Schmolck & Atkinson, 2002). The program accepts 
data entry and then correlates the Q sorts. Factor 
analysis was conducted to show how participants 
grouped according to their sorts. Centroid factor 
analysis was conducted as an initial analysis. Then 
manual rotation was used for more focused factor 
iteration. Participants with similar views shared the 


same factor. In this study, the emergent factors 
represented the different ways participants’ Q sorts 
grouped together regarding their views of helpfulness 
of specific items to their learning of course content. 
Participants with similar views shared the same factor 
(Brown, 1980). 

The PQMethod software calculated the reported 
factor scores as well as how closely the factors 
correlated to each other. The software also aided in 
identifying distinguishing statements and consensus 
statements, underscoring ways that the three factors 
were distinct and similar. The resulting representative 
sorts expressed the overall perspectives of each factor. 
Participants’ comments served to clarify or explain 
some of their sorting choices. In addition to the 
statistical analysis, each of the two researchers read the 
students’ comments and explanations of their rankings 
to find quotes that were relevant to the students’ 
choices. The selected quotes were compared by the two 
researchers, and the ones that best explained the 
participants’ ranking of the items were used as a source 
for data triangulation and establishing credibility 
(Fincoln & Guba, 1985). 

Results 

Three factors emerged from the analysis of the 19 
sorts as displayed in Table 1. Correlations marked with 
an X represent defining sorts. Sixteen students, twelve 
females, and four males loaded to one of three factors. 
Factor loadings “are generally considered to be 
statistically significant if they are approximately 2 to 
2.5 times the standard error’’ regardless of sign (Brown, 
1991, Section 5). For this study, statistical significance 
was calculated at 0.334 - 0.422, signifying p < .05 or p 
< .01. Two sorts did not load on any of the factors as 
they were not statistically significant. One confounded 
sort loaded nearly equally on two factors, thereby not 
identifying with any one particular factor. These three 
sorts were not seen as specifically relating to any of the 
three factors. 

Factor 1 represented the perspectives of 13 
students, Factor 2 included two students sharing a 
different point of view, and Factor 3 represented one 
student’s perspective (Table 1). Despite having just one 
sort, the third factor was retained because that sort 
presented a viewpoint quite different from the others, 
presenting a distinct perspective on the helpfulness of 
the different course aspects. Unlike other factor analysis 
techniques, Q is based upon “self-significance,” or 
“importance to me” (McKeown & Thomas, 2013, p. 
49). According to Brown as quoted by Van Exel and de 
Graaf (2005), “Since the interest of Q methodology is 
in the nature of the segments and the extent to which 
they are similar or dissimilar, the issue of large numbers 
so fundamental in most social research is rendered 
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Table 1 


Factor Matrix with an XIndicating a Defining Sort Loading 


Q Sort 

Factor 1: 

Group Centered Learners 

Factor 2: 

Self Centered Learners 

Factor 3: 

Online Learner 

1 

0.5889X 

-0.4311 

-0.1604 

6 

0.5325X 

-0.2716 

0.1212 

7 

0.3564X 

0.1587 

0.0894 

8 

0.7958X 

0.3926 

0.1406 

9 

0.7210X 

0.0033 

0.0265 

10 

0.6227X 

-0.0842 

0.1156 

11 

0.5210X 

-0.2400 

0.0840 

14 

0.7079X 

-0.2667 

-0.1654 

15 

0.5626X 

0.2932 

0.3759 

16 

0.3831X 

-0.0555 

0.2235 

17 

0.6255X 

0.2554 

0.1616 

18 

0.4683X 

-0.0548 

-0.1400 

19 

0.6439X 

0.3194 

-0.2626 

4 

0.0952 

0.5926X 

-0.1938 

5 

0.1514 

0.7739X 

0.1801 

2 

-0.1905 

-0.0760 

0.4101X 

3 

-0.0052 

0.1200 

-0.0586 

12 

0.0743 

0.2886 

-0.0089 

13 

0.4821 

0.4308 

-0.1364 


relatively unimportant” (p. 5). As Q methodology “aims 
to establish the existence of particular viewpoints” 
(Watts & Stenner, 2012, p. 73), the participants in this 
study expressed three clear perspectives, and 
researchers maintained the three factors that emerged in 
order to highlight those distinct views, even though 
expressed by just one or two participants. These 
participants’ sorts met the threshold for statistically 
significant loadings and provided unique insights for 
the course developer and instructor. 

The majority of the students were on Factor 1, 
named group-centered learners. These students learned 
best by working with other students and the instructor. 
They considered the experiential components of the 
class as the most helpful for their learning. They 
indicated that they learned best when they worked in 
groups and shared ideas with fellow students through 
class presentations, group activities and face-to-face 
discussions. In one student’s words, “Coming to class is 
hands-on, and I like the interaction.” Table 2 lists the 
six highest and lowest ranked statements for Factor 1. 

The highest ranked (5 to 3) items for this group 
demonstrated that these group-centered students 
expressed a preference for in-class tasks such as 
modeling the different second language methods 
studied in class, which was a team project. 
Additionally, presenting the interactive mini-lessons 
they designed as well as observing their colleagues’ 
presentations of their mini-lessons was very helpful to 
them. They also benefitted from discussing different 


topics during class and being engaged in different class 
activities, most of which were done in groups. One 
student stated, “I feel that I learned a great deal from 
the class discussions and activities. I was able to grow 
as a teacher by hearing other people’s ideas and 
strategies.” Among the least helpful, or lowest ranked 
(-5 to -3) activities for Factor 1 were working alone and 
reading assigned articles. Also ranked low were 
researching and writing the final paper, all independent 
learning activities. One learner wrote, “I would rather 
sit in a classroom and learn from my peers.” 

Further clarifying this perspective, a student 
responded to the follow-up question, “Interaction and 
shared stories are the best way to learn methods. 
Hearing other hits and misses helped considerably.” 
Another student wrote, “I really liked all the 
presentations. I’m more of a hands-on learner and being 
able to become a part of my peers’ presentation created 
a lasting impression. It added to the reading. It made the 
reading come alive.” 

Factor 2, termed self-centered learners, were two 
students who presented a perspective that differed from 
the group-centered learners. Unlike their colleagues 
loading to Factor 1, the self-centered learners viewed 
working alone as helpful, and they focused heavily on 
their individual performance in class and events that 
benefitted themselves rather than being mutually 
beneficial to others also. Their work was driven by the 
desire for a good grade, and they also gave the item 
feedback from instructor the highest rank (5). Table 3 
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Table 2 


Factor 1, Group Centered Learners: Most and Least Helpful Aspects and Activities 


Rank 

Item # 

Statement 

Z score 

5 

21 

Observing others' presentations 

2.173 

5 

6 

In-class activities 

1.736 

4 

24 

Classroom discussions 

1.613 

4 

29 

Presenting micro-teaching lesson 

1.242 

3 

22 

My desire for a good grade 

1.133 

3 

23 

Presenting and demonstrating method for class 

0.913 

-3 

5 

Doing research for final paper 

-1.009 

-3 

13 

Writing personal philosophy-final paper 

-1.211 

-4 

2 

Online discussions 

-1.215 

-4 

33 

Methods quiz 

-1.533 

-5 

32 

Reading assigned articles 

-1.544 

-5 

30 

Working alone 

-2.188 


Table 3 

Factor 2, Self-Centered Learners: Most and Least FLelpful Aspects and Activities 

Rank 

Item # 

Statement 

Z score 

5 

22 

My desire for a good grade 

1.957 

5 

7 

Feedback from instructor on quiz and assignments 

1.688 

4 

35 

Preparing micro-teaching lesson 

1.524 

4 

6 

In-class activities 

1.404 

3 

29 

Presenting micro-teaching lesson 

1.389 

3 

18 

Receiving feedback from colleagues on micro-teaching 

1.270 

-3 

14 

Preparing method presentation and handouts 

-0.852 

-3 

23 

Presenting and demonstrating method to class 

-1.120 

-4 

34 

Previous education courses 

-1.136 

-4 

33 

Methods quiz 

-1.270 

-5 

25 

Classmates' methods handouts 

-1.688 

-5 

26 

Working in small groups or with a partner 

-2.091 


lists the highest and lowest ranked sort items for Factor 2. 

The majority of their highest ranked items (5 to 3) 
showed that they were focused more on the preparation 
and presentation of the assigned mini-lesson, primarily 
individually initiated tasks and the major course 
assignment. They felt that feedback from both their 
peers and the instructor was helpful to their learning, 
and providing feedback to others was not ranked 
highly. They benefitted from working independently to 
prepare and present their micro-lessons as well as 
hearing the feedback, which was for completing and 
enhancing their own assignments. The demographic 
data showed that these two students were quite different 
from each other in that one was male, the other female; 
one was undergraduate, the other graduate; and one had 
worked with ESL learners and had taken several other 
TESOL courses, while the other had almost no 
experience with ESL students and this was his first 
TESOL course. As one learner put it, “I think working 
in small groups or partners. . .becomes too much.” 


The lowest ranked statements for these two 
students (-5 to -3) showed they felt that working with a 
partner or in groups was least helpful to their learning. 
One specific online assignment required group 
feedback be provided with participants observing a 
particular timetable. To ensure that each group member 
would have time to respond to all other group members, 
the recommended time frames were liberally assigned. 
A comment from one self-centered learner confirmed 
that the individual preferred learning on his own, even 
within the liberal parameters of the online tasks. That 
Factor 2 participant explained, “The online discussion 
was a hassle. Group work is a practice for team¬ 
building and should NOT be used on adults.” Another 
of their lowest ranked (-3) items referred to one of the 
assignments, team preparation and presentation of one 
of the methods for teaching ESL. These two students’ 
sorts indicated that they didn’t think preparing for the 
methods presentation, doing the presentation, or reading 
their fellow students’ handouts (-5) on the different 



Collins and Angelova 


TESOL Methods 254 


methods were helpful to their learning. The 
explanation that one of them gave was: “The methods 
activity wasn’t as useful because I cannot see ever 
applying a single method in the way we learned them,” 
again, focused on individual preference or need. 

Factor 3, the online learner, represented only one 
student’s viewpoint, a female who taught elementary 
level students and had taught fewer than nine ESL 
students in her classroom in the past two years. Her 
opinion of the helpfulness of the different aspects and 
activities was quite distinct from the rest of the class. 
This student ranked the online discussions very high (5) 
and valued what she had learned from previous 
education courses (5). A number of other courses in the 
TESOL endorsement program were online. In addition, 
she ranked high (3) being able to access additional 
resources provided on Blackboard learning 
management system (Table 4). 

Table 4 also lists the six aspects and activities that 
the online learner considered least helpful to her 
learning. These indicated that coming to class regularly, 
presenting the mini-lesson, observing others’ 
presentations were not as helpful to her as the online 
aspects of the course. Most of these lowest ranked items 
required that the students be together on campus to 
complete them. Her comment was, “I prefer online 
classes. . .” 

Discussion and Implications 

The results from the factor analysis showed three 
distinct views on the helpfulness of the different course 
aspects and activities to students’ learning of TESOL 
methods. Three students did not load distinctly on any 
of the factors. The sorts of two of the students did not 
load on any of the factors as they were not statistically 
significant. The other student’s sort was confounded 
because it loaded nearly equally on two factors, thereby 
not identifying with a factor. These three sorts were not 
seen as relating to any of the three factors as they did 
not share the perspectives of their colleagues. The 
researchers noted that all three of the nonloaders had 
limited prior experience in the TESOL field. This was 
one of the first TESOL courses they had taken, and they 
had worked with fewer than nine ESL students in the 
last two years. For two of them this course was not a 
requirement. Researchers also observed that they did 
not represent the typical population of students who 
enroll in this course, usually taken as a program 
requirement close to the end of the university’s TESOL 
endorsement program. This led researchers to consider 
the possibility that this narrower TESOL background 
(compared to most classmates) may have influenced 
their perceptions of the course activities, setting them 
apart from their colleagues. A comment from one 
nonloader also illustrated a perspective different from 


colleagues: “Working alone and working in a small 
group are very different, but I would leam either way.” 
Others in the class had more specific learning 
preferences regarding interaction with colleagues. 
Another nonloader commented, “The text book is well 
written and easy to understand” and ranked high 
“Reading Larson-Freeman methods book,” an item 
ranked very neutrally by others in the class having more 
TESOL background experience and knowledge. These 
kinds of comments also separated these nonloaders 
from the rest of the class and may have been a 
reflection of their more limited background in TESOL 
and experience with English language learners. 

Sorts and comments of the 13 group-centered 
learners expressed that they learned better through the 
interactive aspects of the course. This led researchers to 
consider that the experiential aspects of the course were 
generally helpful to the majority of students regardless 
of amount of prior TESOL coursework and experience 
working with ELLs. The results of Factor l’s view 
coincided with earlier findings by Gandara, Maxwell- 
Jolly, and Driscoll (2005) that individuals preparing to 
work with ELLs preferred to learn effective TESOL 
strategies by observing other teachers, thereby 
confirming that the interactive parts of the course were 
positive aspects in the syllabus for a high proportion of 
these students. 

The group-centered learners also gave a high rank 
to in-class activities and class discussions, which 
centered upon modeling effective TESOL techniques. 
This finding, too, concurred with research indicating 
that teachers cite learning about second language 
instructional techniques as what they most want to learn 
(Gandara et ah, 2005; Karabenick & Noda, 2004) in 
order to enhance their confidence in working with 
ELLs. 

As experienced ESL teachers themselves, the 
researchers also considered this course from the 
perspective of learning in a second language. This 
course intentionally incorporated activities in which 
students carried out a variety of tasks designed to model 
and allow them to experience learning through TESOL 
instructional techniques based heavily upon interactive 
tasks and carried out in a rich context. Many of the 
students in this class learned English as a second 
language themselves, a fact that may have contributed 
to why some of these “group centered learners” might 
have felt they learned better from these face-to-face 
interactive activities more so than lower ranked, less 
contextualized learning tasks such as reading and 
writing about the content or discussing their ideas 
online. Interactive tasks also highly engage all 
participants and offer information and feedback to all 
those participating in them rather than being directed 
more toward one person. Researchers felt that 
collectively, it was possible that students recognized 
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Table 4 


Factor 3 Online Learner: Most and Least FLelpful Aspects and Activities 


Rank 

Item # 

Statements 

Z scores 

5 

2 

Online discussions 

1.815 

5 

34 

Previous education courses 

1.815 

4 

18 

Receiving feedback from colleagues on micro-teaching 

1.452 

4 

22 

My desire for a good grade 

1.452 

3 

8 

Resources, announcements posted to Blackboard Learn 

1.089 

3 

4 

Methods summary paper 

1.089 

-3 

20 

Coming to class regularly 

-1.089 

-3 

11 

Instructor's availability to explain-in person, e-mail, phone 

-1.089 

-4 

5 

Doing research for final paper 

-1.452 

-4 

24 

Classroom discussions 

-1.452 

-5 

21 

Observing others' presentations 

-1.815 

-5 

29 

Presenting micro-teaching lesson 

-1.815 


these activities as appropriate for ELLs, which gave the 
tasks a feeling of being familiar and inside their 
comfort zone. 

The views expressed by the two self-centered 
learners on Factor 2 differed from those of the group- 
centered learners in that the self-centered learners were 
driven by external motivation, that is, their desire for a 
good grade guided them in their choices of most helpful 
activities. In fact, their other highest ranked item was 
feedback from the instructor - the one who ultimately 
assigns the grades. Receiving feedback is less active in 
nature than interactive tasks, which seemed to fit with 
the overall perspective of these Factor 2 learners. 

These students ranked high their experience with 
mini-lessons presentations, the most heavily weighted 
course assignment which included the most extensive 
feedback, and, ultimately, their grade, which they 
deemed an important feature of the course. This was 
seen as reflecting their self-centered learning nature. 
Unlike the group-centered learners, these self-centered 
learners preferred to study alone, and they focused 
mostly on their personal performance in class, which 
could explain why they ranked feedback from their 
instructor and their mini-lessons the highest (5). 
Researchers wondered if these students may have 
thought that taking into consideration the instructor’s 
feedback, particularly on the heavily weighted mini¬ 
lesson, would ultimately help them achieve a better 
grade. The students from the second factor were more 
individualistic learners as opposed to the collectivist 
team player type of learners from the first group. They 
preferred to study alone and to present their own work 
rather than collaborate with the others, but they also 
wanted to have the instructor’s and peers’ feedback so 
that they could be successful. 

The self-centered learners did not find the 
assignments related to learning about or applying the 
different methods helpful to their learning. One possible 


explanation of this low ranking in Factor 2 might be 
that these two students did not find the group activities 
to be helpful since the format of the methods 
assignment was to work and present with a partner. 
Nevertheless, researchers found this attitude toward the 
methods project to be a bit distressing since this course, 
“TESOL Methods and Materials,” had such a strong 
basis in exploring different language instruction 
methods in order to identify one’s own preferred 
methods. Yet, preparing, presenting and receiving 
feedback on the micro-lesson, which they planned and 
presented individually, helped them learn. This led 
researchers to consider that in the future, the course 
design might need to indicate more explicit connections 
between the assignments related to learning of different 
methods and classroom instruction by redesigning that 
aspect of the course. 

These self-centered learners did not find the online 
discussions to be helpful either, which again might 
reflect their view of group work. To the researchers, 
these two students seemed to be very traditional, 
preferring transmissive instruction in their style of 
learning. They appeared to prefer a teacher-centered 
classroom in which they communicate mainly with the 
instructor whose feedback they value highly, receive 
feedback from colleagues, and learn content through 
independent work. 

The only student on Factor 3, the online learner , 
expressed often cited reasons students mention for 
favoring online courses such as a busy schedule and 
preference for working alone rather than doing group 
projects. (Brown & Green, 2003; Carter, 2004; Harlen 
& Doubler, 2004). This student did not think observing 
others’ mini-lesson presentations was helpful to her 
learning. This view could perhaps have been influenced 
by the fact that these were essentially in-class activities. 
She valued her previous education classes, a number of 
which were online, which may be based upon her 
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preference for working independently rather than 
collaborating with colleagues. 

Factor 2, the self-centered learners, and Factor 3, 
the online learner, ranked receiving feedback among 
their top three items, indicating it was very helpful to 
their learning. These views underscored what was for 
them a strength in the course design since, in addition 
to traditional instructor feedback on their mini-lessons, 
students received three points of feedback from every 
classmate. The usefulness of feedback in the learning 
process is documented as far back as Bruner (1973) 
when he outlined the benefits of scaffolding learning 
with constructive feedback to guide learners to carry 
out a task independently and effectively. The course 
designer’s intent was to make available a variety of 
perspectives and depth of feedback offered to each 
student. The helpfulness noted through the sorting 
activity confirmed that this was a valuable aspect of the 
course. 

Those loading to factors 2 and 3 also appeared to prefer 
to work independently, although in different contexts. 
Factor 2 students appeared to be more externally motivated 
and found in-class activities to be helpful, while the Factor 3 
perspective was more focused on the independence and 
flexibility allowed by the online learning environment. 
Researchers saw these perspectives as underscoring the 
importance of maintaining balance between collaborative 
and independent learning opportunities in the course 
structure. Such differences in learning preferences may also 
indicate that more choices in format or method of 
completing required course work are needed in future 
iterations of the course. 

It is worth noting that 17 of the 19 participants in this 
course (13 on Factor 1, one on Factor 2, one Factor 3 
student, and two of those not loading to a factor) did not 
view writing about their personal TESOL teaching 
philosophy as helpful to their learning. Researchers 
observed that the majority of the students were experienced 
teachers, and these perspectives and student comments 
brought the researchers to consider that possibly students 
had already written about their teaching philosophy in 
previous courses. Since the teaching philosophy paper was a 
major course assignment and 17 students ranked it very low, 
researchers turned to the students’ explanations of their 
sorting decisions for clarification. A Factor 1 student 
confirmed researchers’ thoughts by stating, 

As a current educator, it seems a bit pointless for 
me to discuss my personal philosophy regarding 
teaching. I already did this in undergrad and 
continue to do so every day I teach. To record that 
data for someone else no longer seems to be a 
beneficial learning experience for me. 

The Factor 3 online learner commented, “I do 
many research papers - I do not [get] much out of 


them.” This overwhelming negative view of the 
philosophy paper led the course developer to consider 
restructuring this final task for future implementations 
of this course or reviewing the context in which it is 
presented. 

In order to further ground this work more firmly in 
established and accepted theory, researchers (who 
collectively had designed and taught the course) 
compared the foundation of the course design and 
participants’ responses and comments to Merrill’s 
(2002) “First Principles of Instruction,” which outline a 
complete learning cycle. These features are common to 
various instructional design theories, and if any are 
missing, “learning will be negatively impacted” (Frick 
et ah, 2010, p. 116). These five principles, as stated by 
Merrill (2010) are: 

(a) Learning is promoted when learners are 
engaged in solving real-world problems, (b) 
Learning is promoted when existing knowledge is 
activated as a foundation for new knowledge, (c) 
Learning is promoted when new knowledge is 
demonstrated to the learner, (d) Learning is 
promoted when new knowledge is applied by the 
learner, (e) Learning is promoted when new 
knowledge is integrated into the learner's world (p. 
44-45). 

Researchers found direct parallels between these 
features of an effective course design and the course 
under study, but they wanted to compare students’ 
views with these principles as well. After collecting and 
analyzing students’ perspectives on the course, 
researchers investigated how students’ rankings of 
course tasks and comments coincided with these 
essential features of an effective instructional design. A 
review of all factors showed students’ perspectives of 
the course design coincided readily with principles of 
instruction presented by Merrill (2002) as essential to 
instruction. 

The generally high rankings given mini-lessons 
and other class tasks showed that students found them 
helpful in solving real classroom problems and 
demonstrating new knowledge. One self-centered 
learner even stated, “It was obvious that the interactive 
mock lesson was very helpful in implementing a real- 
world application of the lessons used in a classroom.” 
Factor 3 (online learner ) ranked activating previous 
knowledge high (5). Students acknowledged that the 
new learning was applied directly in their mini-lessons. 
As noted by another group-centered learner: 
“Presenting the micro-teaching gave me a dry run of 
applying our methods ... The feedback that I received 
from my colleagues allowed me to refine my 
approach.” Finally, students indicated that they were 
ready to introduce the methods into their classrooms. 
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As a self-centered learner said: “I really appreciated 
getting legitimate feedback which I felt will personally 
help me reflect and improve my teaching.” A group- 
centered learner noted, “I gathered a plethora of 
teaching materials and activities for my kids.” All of 
this reinforced to the researchers that the majority of the 
in-class work was beneficial to the students’ learning 
and overall, Merrill’s (2002) First Principles of 
Instruction had been included in the course design. 

After being taught more than 20 times and 
evaluated through the university’s traditional Likert 
scale feedback, researchers uncovered students’ distinct 
views of which aspects of the course were most helpful 
to each of them. It must be noted that this study has its 
limitations as it is a single case. However, Q 
methodology was designed for small numbers of 
participants and even single case studies (McKeown & 
Thomas, 2013). Also, the use of Q methodology helped 
the researchers find out what all students agreed upon 
by identifying the consensus statements for all three 
factors. The results showed that nearly all students 
thought that the two textbooks used for this course were 
not very helpful. This led the researchers to wonder if 
there was something about the textbooks that limited 
their helpfulness or if the way they were incorporated 
into the lessons made them less helpful than other 
aspects of the course. Future implementations of this 
course might be enriched if varied ways of using the 
texts were tried or students were questioned more in 
depth about their perspectives regarding the texts. It is 
also possible to change the textbooks for different ones. 
Additional studies using the same methodology should 
be done in the future after making the suggested 
changes based on the results from this study in order to 
find out if these changes would be considered beneficial 
by the students. 

In conclusion, using Q methodology for this study 
helped the researchers/course designer/instructors to 
examine the different views students held about the 
helpfulness of different aspects of the course and 
activities completed in the class when following a 
blended face-to-face and online design. The majority of 
the students seemed to find the hands-on activities, 
mini-lesson presentations, in-class discussions and 
feedback from their instructor and colleagues to be 
most helpful to their learning. Many did not feel that 
the online component of the course was as helpful to 
their learning. One specific activity the students felt 
helped them the most in learning about different 
methods and approaches to teaching ESL were the 
lesson demonstrations. 

More advanced uses of technology would be 
involved in redesigning the course to be totally online, 
as the other courses in the TESOL endorsement 
program are, and may need to be considered. As the 
course designer considers whether an online format 


would be feasible for this course, it seems that 
incorporating the mini-lesson presentations into an 
online course could be challenging. To achieve this, 
students could possibly videotape themselves teaching, 
perhaps in an isolated setting, in which case they would 
not be experiencing a classroom environment with their 
fellow students acting as ESL students. Thus, the 
teaching would be less authentic. It would be more 
difficult, perhaps impossible, to have a discussion 
immediately following the presentation and receive 
feedback, course aspects which students ranked as very 
helpful to their learning. One solution might be to 
restructure this class but continue as a hybrid class to 
accommodate more types of learners with readings, 
activities and discussions online and mini-lesson 
presentations with feedback in a face-to-face setting. 
Giving students more choices in activities or format 
might also support the varied learning preferences of 
different students. 

In this “post method era,” results also seemed to 
indicate that several assignments might benefit from 
revision. Some students appeared to think that it was 
not necessary to give methods presentations and take a 
quiz on the same material. Perhaps replacing the quiz 
with a different way of assessing knowledge and 
understanding of the different methods and approaches 
is in order. In addition, it seems that the final paper on 
the teaching philosophy could also be replaced by a 
reflection paper on one’s preferred methods for 
teaching ESL students or another assignment to 
synthesize and enhance the application of different 
methods of second language instruction. The course 
might be enriched by adding case studies of ESL 
lessons and experiences in ESL classrooms which could 
offer additional hands-on experience, the experiential 
component of the course that so many of the students 
considered most helpful to their learning. Finally, the 
instructors might need to re-examine the use of the 
textbooks and technology for this course. 

This study demonstrated how instructors can 
examine students’ different views of the activities that 
helped them the most in learning the content of a 
course. As noted by Jurczyk and Ramlo (2004) and 
Ramlo (2012b), clearer insights into student viewpoints 
on learning can be captured through Q than through 
traditional Likert scale course evaluations. 

Evaluations at the university in this study consist of 
Likert scale surveys, rating items from 1 (least 
effective) through 5 (most effective). This allows 
students to rate different aspects of the course and the 
professor, although no items directly address student 
learning preferences. The resulting score is an average 
of the ratings of all students, which can often provide a 
somewhat inaccurate picture of the course (Jurczyk & 
Ramlo, 2004). For example, if half of the students rate 
the instructor’s communication skills as 5 and half rate 
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these skills as 1, the final report rating would be the 
average of those scores, or 3. This score would indicate 
that the students found the instructor to be an average 
communicator. Yet that is far from what their 
individual scores indicated, and students’ views were 
not fully expressed or heard. 

Using Q methodology to study students’ opinion of 
the helpfulness of different activities and projects in the 
course gave the instructor and researcher a chance to 
understand which activities students found most and 
least helpful. Such was the case with the final paper 
about students’ personal teaching philosophies and the 
different tasks surrounding the ESL instruction methods 
which had never been brought to the 
researcher’s/course developer’s attention before. 
Specific feedback such as what was provided through 
this Q study is much more valuable to faculty in making 
decisions about the content and format of their classes 
than the traditional Likert scale evaluations that 
produce global observations and that are frequently 
used at the end of each course (Frick et al., 2010). 

Differences among students’ perspectives of 
courses which only use Likert scales for evaluation 
might go undetected if just final average scores are 
reported to the university administrators and the course 
instructor. Additionally, reasons for these differing 
scores might never be brought to the surface, denying 
the opportunity to understand better how the course 
might be more closely matched to the needs of all 
students, as using surveys often does not permit 
individual voices to be represented (Lecouteur & 
Delfabbro, 2001; Ramlo, 2008). 

As a possibility for other course developers and 
instructors, the researchers suggest they design Q 
studies to evaluate their own courses. Their sorts and 
research design could be based on the questions they 
have about their courses and reflect different aspects of 
the course as well. Their Q sort items would be unique 
to their specific course, would provide insights about 
their students’ perspectives of the classes they teach, 
and would allow them to make informed decisions for 
their future course implementations, as “this research 
enables improved student input regarding teaching and 
learning.” (Jurczyk & Ramlo, 2004, p. 3). 
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Appendix 


List of sort items used in study: 

1. Reflection on micro-teaching 

2. Online discussions 

3. Rubrics 

4. Methods summary paper 

5. Doing research for the final paper 

6. In-class activities 

7. Feedback from instructor regarding quiz and assignments 

8. Resources and Announcements posted to Blackboard Leam 

9. Interacting with instructor outside of class time or during breaks 

10. Giving feedback to colleagues regarding their micro-teaching 

11. Instructor’s availability to explain what I did not understand (in person, e-mail, online, in class, phone call, 
etc.) 

12. Personally choosing skill for micro-teaching 

13. Writing personal philosophy paper (final paper) 

14. Preparing method presentation & handout 

15. Instructor’s lectures, PowerPoints 

16. Reading the Larson-Freeman Methods book 

17. Lesson plan template for micro-teaching 

18. Receiving feedback from colleagues on micro-teaching 

19. Discussing with peers how to present method 

20. Coming to class regularly 

21. Observing others’ presentations 

22. My desire for a good grade 

23. Presenting & demonstrating Method for class 

24. Classroom discussions 

25. Classmates’ Methods handouts 

26. Working in small groups or with a partner 

27. Reading H. Douglas Brown text book 

28. My personal interest in the topic of the course 

29. Presenting micro-teaching lesson 

30. Working alone 

31. Syllabus 

32. Reading assigned articles 

33. Methods quiz 

34. Previous education courses 

35. Preparing micro-teaching lesson 



